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ABSTRACT 



This paper focuses on the Interactive and Distance Education 
Assessment (IDEA) laboratory, an assessment system developed by psychology 
students and faculty at Rensselaer Polytechnic Institute (New York) that 
incorporated learning styles, teaching styles, and other measures of 
individual difference into the evaluation of interactive and distance 
learning. Learning styles and their relevance to research on distance 
education are discussed, and the following five individual difference 
measures are described: self-efficacy; positive affect; locus of control; the 
Learning Style Inventory (LSI) ; and the Work Preference Inventory. 

Development of two versions of the IDEA system (i.e., a World Wide Web 
version and an Intranet version created in LotusNotes) is summarized. 
Preliminary results of a pilot study that incorporated learning style 
measures with the technology developed to deliver evaluation instruments are 
presented; satisfaction surveys and the LSI were delivered via the IDEA 
system to employees of an international company who participated in a 
distance education course on accounting and managerial finance topics. A copy 
of the satisfaction survey is included. (Contains 11 references.) (DLS) 
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The feedback systems currently used by many instructors are much like autopsies. Once the 
class is over, they attempt to discern what went wrong. This post-hoc process will become 
increasingly unacceptable as universities and corporations compete for the education of the 
global market. One solution is an innovative evaluation system, developed by Psychology 
students and faculty at Rensselaer, which provides a high quality of service to students in 
distance education, interactive learning, and studio-based courses. The Interactive and 
Distance Education Assessment Laboratory (IDEAL) was designed to provide quick-time 
feedback to instructors, allowing them to adjust their delivery during the class period. This 
technology provides the feel of a small classroom-type environment to students, even those 
physically located across the globe. It may also be used, in conjunction with an evaluation 
team, as a way to reduce the time-consuming chores of test construction and grading, 
allowing more time for teaching and assisting student learning. 



Introduction 

In a small classroom, an instructor can adjust the content, pace, and emphasis of course 
material depending on the feedback provided by the students through verbal and non- 
verbal cues. A puzzled look or a pause in note taking may prompt an instructor to restate a 
point or to give an illustrative example. In a large classroom, even the best instructors have 
difficulty determining whether some or all of the students are "getting it" during the course 
of a lecture, demonstration, or discussion. In a distance education-based course, the ability to 
receive on-going feedback from students during the delivery of material ranges from 
extraordinarily difficult to nearly impossible, depending on the media that is used. 

However, an innovative evaluation technique is nearing its completion, whereby instructors 
can receive feedback and offer a small classroom-type environment to students who are 
physically situated across the globe. This technology can be illustrated in the following 
scenario in which an instructor teaches a course via satellite to 200 students across five 
continents: 

Following a brief lecture, the instructor directs students to a website containing a 
brief quiz or "concept check"; 15 items are constructed to measure comprehension of 
key concepts and 5 items assess attitudinal measures, such as the perceived 
relevancy, difficulty, and efficiency of the lecture. The students point-and-click 
through the questionnaire and submit their responses. 

Within seconds, all 200 responses are immediately collected, sorted into folders 
based on classroom location and student, dropped into a spreadsheet, analyzed by a 
macro, and the results are printed and electronically delivered to the instructor. 

Within minutes, the instructor has descriptive statistics for each item and can 
^ determine which topics need further clarification, and can estimate the current level 

QQ of student satisfaction with the discussion. For example, perhaps the majority of 
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students in North America scored well on comprehension, but students in the Korea 
site did not (indicating a possible cultural difference that the instructor may resolve 
by using terms less imfamiliar to the Asian students). Perhaps a majority of students 
across all sites scored low on a particular test item, indicating that the topic 
represented by that item needs further clarification. The System Administrator, who 
has discussed the evaluation items with the instructor apriori, may provide nearly 
any descriptive or inferential statistic, classification, or categorization the instructor 
may desire. For example, the instructor may wish to know if students with the 
highest scores on a particular topic are also those tending to be strong in a particular 
learning style and whether this holds true across both American and European 
students. 

The technology imderlying this evaluation technique has existed for some time. Indeed, the 
means by which education is delivered has not been constrained by technology or human 
creativity. It has, however, been constrained by psychological and social limitations. The 
important question is not can we teach using interactive or distance learning technologies, but 
rather how can we maximize student learning through their use. Our research team has foimd 
that the incorporation of learning styles as a measure of individual differences among 
students, has been useful in explaining student performance in the distance education 
classroom, as well as in interactive learning studios and collaborative classrooms. Our 
solution, therefore, was to construct an assessment system which incorporated learning 
styles, teaching styles, and other measures of individual difference into the evaluation of 
interactive and distance learning. This system, called IDEA (Interactive and Distance 
Education Assessment) is described herein. 



Learning Styles 

The evaluation of distance education has often focused on comparisons between traditional 
and distance formats and on the characteristics of "effective" students and instructors of 
each format. In brief, researchers have found that distance education instruction is perceived 
to be less clearly presented, results in higher test scores of achievement, forces instructors to 
become better organized and prepared, and requires extensive formative evaluation to be 
effective (Souder, 1993). Distance-learning students appear to be more successful, in part, 
because they tend to be older, more motivated and self-disciplined, are more likely to 
possess a college degree, and have expectations for higher grades (Gottschalk, 1996). 

What is often missing from the research on distance education is the inclusion of learning 
styles as a variable of interest. The concept of learning styles, that students have a preferred 
way of using their intellectual abilities (Sternberg & Grigorenko, 1997) has a long history in 
Psychology. Research suggests that students whose learning styles are compatible with the 
style of teaching delivered in the classroom, may perform higher than students whose 
learning styles are not compatible with such an instructional style (Furnham, 1992; Honey & 
Mumford, 1986; Ingham, 1991). Every successful instructor realizes that not all students 
learn best using the same method. Some students prefer a hands-on approach in which they 
can see and feel the material in order to best understand; some prefer to work with 
teammates; others gain knowledge by just reading the text themselves; and still others find 
that real-life examples or stories from their instructors are most informative. The key, 
therefore, is to identify which students favor which particular methods in the form of their 
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learning style. Failure to account for this individual psychological difference may explain the 
ambiguous evaluation results that often result. 

Our previous research (Champagne, Goldberg, Glinert, Breimer, Lim, Moyer, & Turmey, 
1998) included the following five individual difference measures: 

♦♦♦ Self-efficacy, or the confidence one has in being able to succeed at a particular task, 
has been shown to lead to greater persistence on a task (Bandura, 1982), which can be 
demonstrated to result in higher performance. 

♦♦♦ Positive affect (PA) reflects the extent to which one feels enthusiastic, active, and alert. 
High energy, full concentration, and pleasurable engagement describe an individual 
with high PA, while sadness and lethargy describe one with low PA. Negative affect 
(NA) is a general dimension of subjective distress and unpleasurable engagement that 
subsumes a variety of aversive mood states, including guilt, disgust, anger, contempt, 
nervousness, and fear. Therefore, one low in NA would be described as calm and 
serene. 

♦♦♦ Locus of Control (Rotter, 1966) measures an individual's expectancy for control of 
reinforcement. Individuals with an internal locus of control believe that events are 
due to personal, stable, relatively permanent characteristics, whereas individuals with 
an external locus of control believe that events or consequences are under the control 
of someone else, and are not dependent on their own actions. 

♦♦♦ The Learning Style Inventory (LSI: Kolb, 1976) is a 12-item instrument designed to 
assess how individuals learn and how they deal with day-to-day situations. The 
responses indicate each student's emphasis on four learning modes: Concrete 
Experience (CE), Reflective Observation (RO), Abstract Conceptualization (AC), and 
Active Experimentation (AE). In the CE stage of the learning cycle, an individual 
tends to depend more on his or her feelings, rather than a systematic approach to 
problems and situations. In the RO stage of learning, individuals use patience, 
objectivity, and careful judgement before contemplating action. The AC stage of 
learning involves logic and analysis, rather than feelings, to understand problems and 
situations. In the AE stage of learning, individuals experiment with ways to influence 
or change situations and are concerned with what really works (Kolb, 1976). 

♦♦♦ The Work Preference Inventory (WPI: Amabile, 1987) is a 30-item instrument designed 
to assess the intrinsic and extrinsic motivational orientations of college students and 
working adults toward their work (the college student version was used). Although 
primary scales of the WPI are intrinsic and extrinsic motivation, these scales are 
broken down into secondary scales. These include enjoyment, challenge, outward, 
and compensation. Enjoyment and challenge are representative of extrinsic 
motivation and outward and compensation are representative of extrinsic motivation. 

After two semesters of evaluating several undergraduate courses (e.g.. Physics I, Computer 
Science I, Graph Theory), the LSI and self-efficacy measures were demonstrated to be the 
most useful of the five original measures. For example, in the Physics I course, the 
correlations between two measures of self-efficacy and final grades were higher (r = .42 and 
r = .38) than were the correlations between final grades and an existing measure of 
knowledge of Physics concepts (r = .22). In the Graph Theory course, students who scored 
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higher on the Reflective Observation (RO) learning mode of the LSI, also tended to do well 
on three of the four individual homework assignments (r = .69, p < .05; r = .60, p < .10; r = .63, 
p < .06). Their RO scores were not correlated with any of the team projects for the students. 
This is encouraging since the RO learning mode is characteristic of tendencies to use one's 
own thoughts and feelings to form opinions and to use patience and careful judgement 
before contemplating action. Although such tendencies may be appropriate for individual 
work, they may not be appropriate for teamwork to be completed in a short period. 

The Technology 

In order for students in a distance education course to be able to "point-and-click" their way 
through the learning styles measures and receive quick-time feedback as described in the 
scenario above, the measures had to be posted to a website. This was completed and pilot 
tested by students taking the Physics I course on campus. However, in order to be flexible, 
the evaluation system had to be able to accommodate organizations that have no access or 
restricted access to the Internet. For this reason. Intranet versions of the same measures were 
created in LotusNotes®. The two versions differ only in terms of how the measures are 
received. Students access the website in the Internet version, but students are sent the 
measures via e-mail in the Intranet version. In both cases, students read the instructions, 
point-and-click through their responses, and then hit the "submit" button to send their 
responses back to the IDEA laboratory for immediate analyses. 

Two advantages of this system are the small classroom "feel" provided, and the assurance of 
reliable and valid measures. The instructor can obtain as much information from her 
students across the globe as that obtained in a traditional classroom. The feedback provides 
the instructor with the opportimity to adjust her teaching style to accommodate individual 
differences in learning style, as well as resolve common misunderstandings that may stem 
from differences in language and culture. In terms of reliability and validity, the evaluation 
team provides the validation of the comprehension items to assure the instructor that the 
questions accurately measure what they purport to measure and are consistent across 
students, between classes, and over time. 

The following pilot study incorporated the successful learning style measures with the 
technology developed to deliver evaluation instruments. Since the organization sponsoring 
the distance education course had limited access to the Internet, the LotusNotes® version 
was used. The course is still in progress, but some preliminary results are offered here. 

Preliminary Study 

Participants 

166 employees of an international company participated in the distance education course. 
They were located in Australia, Brazil, Canada, England, Finland, France, Germany, Korea, 
Mexico, Netherlands, Sweden, and the United States. The U.S. employees were located in 
Alabama, Massachusetts, New York, South Carolina, and Wisconsin. Attendees in the course 
ranged widely in age, experience, and job titles, although the majority were middle to upper 
level managers. Participation in the course was voluntary. 
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Procedure 



The course was entitled "Financial Wellness" and consisted of a combination of Accounting 
and Managerial Finance topics. Seven two-hour lectures were delivered live via PictureTel® 
from a New York site over a two-month period. A financial expert (in the form of an upper- 
level financial manager) was present at each site to offer support and to clarify minor points 
during the lecture or after class. 

The instructor began each class by "taking role," whereby each of the 20 sites would check in 
to ensure there were no technical difficulties. Each site answered by pressing "mute off" on 
their remote control, allowing the instructor to hear the reply. Each site was muted unless 
participants had questions to ask of the instructor. Students from each site could view both 
the instructor and a still shot of the white-board that presented relevant financial and 
accounting formulae. Lectures were frequently punctuated with questions from the sites, 
which were answered by the instructor as they arose. 

Following the first lecture, the participants were sent a satisfaction survey (see Table 1) via 
LotusNotes® technology. This survey consisted of 23 items intended to measure student 
attitudes towards the course material, the delivery of the material, the instructor, and the 
"fit" of the course with their career goals. The measure was similar to the course evaluation 
used by students in courses at Rensselaer, but revised to accompany items requested by the 
organization and the instructor. 

Since all participants were familiar with using electronic mail, the survey was delivered in 
the form of an e-mail message to be completed and returned. The results of individual items 
on the survey were delivered to the instructor prior to the next lecture session in order to 
adjust course content and delivery. In addition, the students were encouraged to send 
individual comments to the evaluators about any aspect of the course. 

Following the third lecture, the participants were sent the Learning Styles Inventory (LSI: 

Kolb, 1976) via the same technology. This data has been collected, but carmot be 
meaningfully reported until the final performance measure is given (see below). 

Measures Yet to Be Administered 

The participants will be sent the same satisfaction survey following the fifth and seventh 
lectures. At the conclusion of the course, the instructor will hand out a final examination to 
be completed individually. This performance measure will assess comprehension on the 
topics discussed throughout the course. 



Preliminary Results and Discussion 

Sixty percent of the participants responded to the first satisfaction survey, and the results 
were highly positive. For example, 89% of participants agreed that the course was well 
organized, 73% agreed that their understanding of the financial concepts improved after the 
first session, and 85% agreed that the examples and illustrations were effective. Overall, the 
average of the majority of the items was significantly higher than those reported from 
traditional course evaluations. 
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Of more interest, from a problem-solving point of view, are the more negative results. For 
example, item #12 (see Table 1) stated, "The technology used to deliver this course is 
hampering my ability to learn." Those participants who agreed with this statement also 
tended to find the topics less interesting (r = -.46), felt less motivated (r = -.36), foimd less fit 
of class to career goals (r = -.48), had less interest in the subject (r = -.47), and rated the 
instructor (r = -.48) and class (r = -.55) lower overall (all ps significant at .01 alpha level). In 
addition, eight participants disagreed with the statement, "This course fits well into my 
overall career goals." It wiU be interesting to note if both sets of individuals will have a 
higher dropout rate than those participants who did not feel that the technology was 
hampering their ability to learn or who felt that the course fit their career goals. 



Table 1. Satisfaction Survey Administered to Distance Education Students 



1. The course is well organized. 

2. The course material is being covered too quickly. 

3. My understanding of accounting and finance has improved as a result of the first 
session. 

4. I see that this course will bring measurable value to my job 

5. The instructor presents the subject matter clearly and effectively. 

6. The instructor stimulates my interest in the subject. 

7. The instructor encourages questions and student participation. 

8. The instructor is well prepared and uses class time well. 

9. The instructor uses effective examples and illustrations to clarify concepts. 

10. The instructor has excellent command of the topics. 

11. The language differences are hampering my ability to learn. 

12. The technology used to deliver this course is hampering my ability to learn. 

13. I was knowledgeable about the course topics prior to taking this course. 

14. I am confident that I will do well in this course. 

15. This course fits well into my overall career goals. 

16. I prefer the distance-learning format rather than having an instructor present in the 
room. 

17. I find the topics in this course to be interesting. 

18. I am motivated to do well in this course. 

19. Frequent breaks with local discussion would be helpful to the learning process. 

20. I see the relevance of this course to my current job. 

21 . Overall, I would rate this instructor as . . . 

22. Overall, I would rate this class as . . . 



Note. Statements 1 through 20 use the following response scale: 1 = Strongly Disagree; 

2 = Disagree; 3 = Neutral (choose this response if you can't decide whether you agree or 
disagree, or if statement is not applicable); 4 = Agree; 5 = Strongly Agree. Statements 21 and 
22 use the following response scale: 1 = Poor; 2 = Average; 3 = Good; 4 = Excellent. 



Interestingly, only 3% of respondents preferred the distance-learning format rather than 
having an instructor present in the room. The impact of this attitude towards the technology 
has yet to be examined. There was also a large range of responses to the statement, "I was 
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knowledgeable about the course topics prior to taking this course." These responses will be 
correlated with the performance measure and final satisfaction measures to determine the 
role of experience with attitudinal and ability outcomes. 



Advantages of This System 

Although the focus of this pilot test has been on the flexibility and variety of feedback 
provided by this technique and the role of learning styles, there are many other uses for the 
IDEA system: 

Separation of testing and teaching. A common complaint of instructors is the amount of time 
spent constructing and administering exams. Use of the IDEA system will significantly 
reduce time spent by instructors and their teaching assistants in producing and correcting 
exams. Apriori discussions between the evaluation team and the instructors about the course 
content and structure, the instructor's objectives, and his or her preferred teaching style, will 
provide the basis for erecting an evaluation system that is comprehensive in scope but 
efficient to implement and maintain. This efficiency will shift the burden of testing and 
grading to the evaluators, allowing the instructor to spend more time teaching and assisting 
students. 

Synchronous classrooms. Although the pilot study described here was for asynchronous 
learning, the system is designed to provide evaluation for synchronous distance learning as 
described in the earlier scenario. Regardless of the manner in which the course and feedback 
is delivered, the content of the evaluation would be the same. 

The role of information technology in education. The research and education communities 
have had great success in developing technology-based educational delivery formats (e.g., 
interactive, satellite, and Web-based learning). However, similar progress has not been made 
in understanding how these technological delivery formats interact with psychological and 
social characteristics of the learners (e.g., learning styles, cognitive needs, and preferences in 
learning envirorunents). The strength of the IDEA laboratory is in merging information 
technology with psychology to address the concerns of the emerging global university. 
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